Meeting the User’s Intention in Programming by Demonstration Systems

A Programming by Demonstration or PbD system allows its users to construct procedural knowledge or programs on an abstraction level highly above traditional programming languages. The acquired procedural knowledge can be used to automate repetitive tasks and to enable users to concentrate their efforts on the creative part of their work. The major application areas are interactive systems as graphics editors [Maulsby and Witten, 1993; Sassin, 1994], text editors [Mo and Witten, 1993], hypertext systems [Cypher, 1993], spreadsheets [Spenke, 1993], personal assistant agents [Bocionek, 1995], and robotics [MUnch et al., 1994]. While the use of classical or visual programming languages require an extensive training to enable users to describe procedures, PbD systems support the acquisition of procedural knowledge for interactive systems while users are "manually" solving a task in utilizing the given user interface of such a system. Ideally, the PbD system acquires a new program, e.g. to extract subject lines of email [Cypher, 1993], by observing and analyzing a sequence of actions performed at the user interface of the interactive system, in this particular case a combination of a mail tool and hypertext system. Since the user knows the handling of this interactive system already, no training would be necessary to acquire a new program. Nevertheless, a PbD system with such qualities is hard to develop. One of its major problems is that single actions and action sequences can be interpreted in various ways since there is often not enough information about the user’s goals or intention available. Thus, the PbD system has to consider many possible hypotheses spanning a large search space. Although several machine learning algorithms are available that generalize a concept based on one or more examples, PbD researchers have been quite disappointed by the standard ML algorithms [Witten, 1995]. On the one hand, inductive ML algorithms require a large number of examples that cannot be provided by the demonstrating person. On the other hand, deductive algorithms need a strong domain theory that cannot usually be defined for the interactive system beforehand or limits the flexibility of the PbD system by doing so [Sassin and Bocionek, 1993]. Therefore, most PbD applications consist of various pragmatic assumptions, heuristics, and -as we propose in this paper -different ways of user interaction. Additionally, the ML algorithms are often not suited for the analysis of action sequences and the derivation of procedures with loops and branches [MacDonald, 1991]. New methods have to be developed that enable a PbD system to derive program structures that are compatible with the user’s intention. In order to find and verify such programs with the user’s intention, new methods of visualization and interaction have to be provided that go beyond just showing program code described in a program language [Myers, 1988] or anticipation of actions [Cypher, 1993; Maulsby and Witten, 1993]. The user wants to know what was derived to really trust the derived program. Since the environment of a PbD system is interactive by its nature, users are available during as well as after a demonstration sequence has been completed. Therefore, the derivation power of a PbD system can be enhanced if the user interacts with the system as a teacher. For example, often an action can be interpreted in a large number of possible ways in a particular situation. To solve such ambiguities we propose system-initiated dialogs which let the user choose amongst possible alternatives. In our graphics editor ProDeGE+ hypothesis selection panels list possible interpretations sorted by some plausibility criteria and enable the user to verify or change the automatically derived hypothesis according to the user’s intention. User assistance is also necessary to derive the correct program structure. Our PbD system visualizes program structures with a two-dimensional history board and enables the user to adapt it to the program he or she had in mind by user-initiated interaction. In the following chapter we will discuss the architecture of our PbD system and in particular our ideas for integrating a variety of interaction mechanisms. The architecture is applied to our PbD graphics editor ProDeGE+ [Sassin, 1994Jthat will be described in chapter 3. Chapter 4 will summarize our results and outline future research topics.